REMARKS 

Enclosed herewith is a Substitute Specification in which 
the specification as filed has been amended in various places to 
correct typographical and grammatical errors, and also to add 
section headings. 

In support of the above, enclosed herewith is a copy of 
the specification as filed marked up with the above changes. 

The undersigned attorney asserts that no new matter has 
been incorporated into the Substitute Specification. 

The claims have been amended to more clearly define the 
invention as disclosed in the written description. In particular, 
the claims have been amended for clarity. 

The Examiner has rejected claims 1-5 under 35 U.S.C. 
102(a) as being anticipated by International Patent Application No. 
WO 99/16251 to De Haan et al . The Examiner has further rejected 
claim 7 under 35 U.S.C. 103(a) as being unpatentable over U.S. 
Patent 5,612,743 to Lee in view of U.S. Patent 5,933,535 to Lee et 
al. 

The De Haan et al . reference discloses motion estimation 
and motion- compensated interpolation. This is described in the 
Substitute Specification on page 1, paragraph [0001] , and in 
particular from page 2, paragraph [0004] to page 3, paragraph 
[0005] , and on pages 3 and 4, paragraph [0007] . 
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The subject invention, as claimed, makes use of the motion 
estimation method of De Haan et al . However, this motion estimation 
is only performed in selected blocks in the image. In particular, 
the subject invention includes, as claimed in, for example, claim 
1, "selecting parts of an image frame in which a first video image 
is significantly distinguished from a second video image" . The 
selected parts of the image are then used to determine the 
"parameter sets of two or more motion models". As such, as 
described in the Substitute Specification on page 4, paragraph 
[0008] , the typical complex calculations for the motion estimation 
are only performed for the selected parts as opposed to the entire 
image . 

This process of the subject invention is further detailed 
in claim 2, which claims "dividing a current and a previous video 
image into respective pluralities of blocks", and "evaluating 
deviations between the current and the previous video image block 
by block, taking those blocks as said selected parts in which a 
value of the deviation exceeds a predetermined threshold value". 

Applicants submit that De Haan et al. neither shows nor 
suggests the selection of parts of the video frame that are 
distinguished between a current and previous video image, i.e., 
those block in which a value of a deviation exceeds a threshold 
value, and then performing the motion estimation method using only 
the selected parts/blocks. 
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The Lee patent discloses a method for encoding a video 
signal using feature point based motion estimation, in which, as 
shown in Fig. 3 therein, a current frame signal and a reference 
frame signal are applied subjected to block-based motion 
estimation . 

The Lee et al . patent discloses an object-based video 
compression process employing arbitrarily- shaped features, which 
arguably includes motion estimation, and is included in a computer 
system for implementing the method and apparatus embodying the 
invention. 

Applicants, however, submit that neither Lee nor Lee et 
al. disclose or suggest that the current and reference frame 
signals should be compared to select "parts of an image frame in 
which a first video image is significantly distinguished from a 
second video image", and then subject only the selected parts to 
the motion estimation process. 

In view of the above, Applicants believe that the subject 
invention, as claimed, is neither anticipated nor rendered obvious 
by the prior art, either individually or collectively, and as such, 
is patentable thereover. 
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Applicants believe that this application, containing 
claims 1-7, is now in condition for allowance and such action is 
respectfully requested. 



Respectfully submitted, 




Attorney 

Tel.: 914-333-9611 
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PHNL 000643 

ESTIMATION OF MOVEMENTS IN VIDEO IMAGES 

BACKGROUND OF THE INVENTION 
Field Of The Invention 

[0001] The invention relates to a method of motion estimation in 
video image data, in which m e thod , starting from a first and a 
5 second video image, parameter sets of two or more motion models are 
initially determined, and in which image objects are assigned to 
the motion models. The invention also relates to a device for 
performing the method, a device operating in accordance with the 
method for displaying video images, and a computer program product 

10 for motion estimation. 

[0002] Advances in multimedia techniques have led to the 
development of a multitude of video formats and display standards. 
They are distinguished, inter alia, also by their image rate, i.e.^ 
the number of frames per unit of time. When a video sequence is to 

15 be displayed on a PC or TV display screen, it is necessary to adapt 
to the image rate of the display apparatus. Interfaces suitable for 
this purpose operate by m e ans of conversion methods of a^different 
complexity. The simplest method is to repeat Jor omit]_ frames of 
the video sequence in the display, dependent on the desired image 

20 rate. However, when displaying video data thus treated, unwanted 
artifacts occur. Unwanted display errors occur dependent on the 
ratio of the image rates involved. The display appears to be 
jittery and irregular so that the motions displayed in the video 
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sequence have an unnatural effect. More elaborate methods perform 
an interpolation between consecutive video images, in which an 
algorithm for motion estimation is used, which this algorithm 
initially r e cogniz e s recognizing the displacements of individual 
5 pixels from one image to the other^ and g e n e rat e s generating image 
data therefrom which are temporally present between the images of 
the video sequence . The use of such methods in apparatus for home 
use requires the fundamental algorithms to supply a qualitatively 
high-value image rate conversion and require only a small number of 

10 computations because the digital signal processing electronics in 
apparatus for home use have a limited efficiency. 
[0003] Motion estimation methods of the type described in the 
opening paragraph are not only suitable for image rate conversion^ 
but also for coding and compression in the transmission of video 

15 data, as well as for depth estimation in 3D image data processing, 
and for disparity estimations in stereo images. 

Description Of The Related Art 

[0004] Such a method is proposed in International Patent 
20 Application No. WO 99/16251 , corresponding to U.S. Patent 

6, 385, 245 . It is an efficient, object-oriented method of motion 
estimation in which two or more motion models are used so as to 
describe the displacement of image objects between a current and a 
previous video image. The motion models are determined by parameter 
25 sets from which displacement vectors can be computed. One of the 
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motion models is needed to deal with those image parts which are 
static. The associated displacement vector is thus the zero vector. 
The parameter sets of the other motion models are determined by 
evaluating the match errors of the motion models in the description 
5 of the displacement of image objects between consecutive video 

images. For the interpolation, it is then necessary to segment the 
image data and assign appropriate motion models to the individual 
image objects. The result of the segmentation is separate objects, 
i.e.^ image parts which perform a similar or comparable 

10 displacement from the previous to the current video image. 

[0005] The known motion estimation method is an efficient 
alternative to the otherwise conventional block-oriented method 
because the number of independently movable objects is small in 
normal video sequences and, consequently, only a correspondingly 

15 small number of motion models is to need be processed. A small 

number of computations results therefrom, which renders the method 
universally usable, also for home use. 

SUMMARY OF THE INVENTION 
2 0 [0006] The fundamental object of the present invention is to 

further improve the known motion estimation method and^_ 

simultaneously^ further reduce the complexity. 

[0007] An important step in the motion estimation is the 

determination of the parameter sets for the motion models. In the 
25 known method, the parameter sets are combined to vectors. For each 
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motion model, a parameter set is selected from a quantity of 
candidate vectors in accordance with a selection criterion. The 
selection criterion consists of the evaluation of a match error. 
This is computed as the sum of absolute differences of individual 
5 motion-compensated pixel intensities between the current and the 
previous video image, while a displacement vector in accordance 
with one of the candidate vectors is used for compensation. An 
essential problem is that it is not clear in advance which motion 
model is to be assigned to which image area and with which 

10 parameter set. The known method is performed in such a way that the 
above-mentioned selection criterion is initially used with all 
motion models for the same image areas. Then, without an assignment 
being fixed, the best fitting parameter sets are selected. 
[0008] In accordance with the above-mentioned envisaged object, 

15 a further reduction of complexity in a motion estimation method of 
the type described in the opening paragraph^ is achieved in that 
only parts of the image area are taken into account when 
determining the parameter sets. A problem then is that 
corresponding parts of the image area are to be selected in an 

2 0 appropriate manner so that displacements between the video images 

are as completely captured as possible. According to the invention, 
only those parts of the image area are thus taken into account for 
determining the parameter sets, in which the first video image is 
significantly distinguished from the second video image. 
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[0009] Such distinctions are a clear indication that there is a 
displacement at the corresponding locations. Image parts for 
determining the parameter sets can thereby be selected very easily 
without initially having to know more precise motion data. 
5 Moreover, it is avoided that stationary image parts are processed 
when determining the parameter sets for the motion models, for 
which stationary parts motion compensation is useless and_^ 
therefore^ unnecessary. In fact, the parameters must be determined 
only in the non-stationary image parts. 

10 [0010] Since the selection criterion is only used for parts of 
the image area, the number of required computations is greatly 
reduced so that the overall motion estimation is accelerated. Based 
on the fact that only some hardly moving objects are displayed in 
typical video sequences, it is sufficient^ under normal 

15 circumstances^ to limit oneself to a corresponding number of 
"interesting" points in the video image when determining the 
parameter sets. 

[0011] The "interesting" image areas are suitably determined in 
that deviations between the video images are evaluated block by 

20 block, taking those blocks for determining the parameter sets into 
account in which the value of the deviation exceeds a predetermined 
threshold value. The image area is thus divided into individual 
blocks whose size should be dimensioned in such a way that the 
parameter sets can be determined by means of individual blocks . The 

25 deviations between the current and the previous video image may be 

ST14GOA0 . RED 5 



PHNL 000643 

determined, for example, by forming the absolute differences of the 
pixel intensities^ each time^ within the individual blocks. The 
result is a positive number so that it can be easily ascertained^ 
by comparison with a predetermined threshold value^ whether there 
5 is motion or no motion in the associated image area. When 
determining the parameter sets, the method according to the 
invention is limited to those blocks in which a given distinction 
between the two video images can be recognized on the basis of the 
pixel intensities . 

10 [0012] This method has the additional advantage that the 

threshold value can be determined on the basis that the number of 
image areas taken into account for determining the parameter sets 
is limited to a predeterminable value. Since the overall method is 
to be performed in real time for the motion estimation, it should 

15 be ensured that the number of computations remains below a fixed 

maximum value. It is thus possible to adjust the threshold value in 
the case of repeated use of the method according to the invention 
in such a way that the data processing time remains uncritical. 
[0013] Practice has prov e d proven that it may be advantageous to 

2 0 take into account those parts of the image area for determining the 
parameter sets, in which motion was determined in previous video 
image data of a sequence of video images. A higher temporal 
consistency in the motion compensation is obtained in this way. 
[0014] For performing the method according to the invention, a 

25 device for motion estimation in video image data is suitable, which 
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this device compri ses comprising a digital image memory in which a 
first and a second video image can be stored, and means for 
determining parameter sets of two or more motion models in 
accordance with a selection criterion. The device according to the 
5 invention comprises means for block-wise evaluation of the 

deviations between the current and the previous video image^ and 
for selection of those blocks for use of the selection criterion, 
in which the value of the deviation exceeds a predeterminable 
threshold value. Such devices may be used, for example, as 

10 components in television and video apparatuses. The digital image 
memory of the device according to the invention^ need not 
necessarily have a sufficient capacity for recording the first and 
the second video images simultaneously. The consecutive storage of 
the respective images is sufficient for the method according to the 

15 invention. 

[0015] Devices for displaying video images such as, for example^ 
televisions, monitors^ etc., comprising a digital image memory in 
which video image data can be stored, and electronic means for 
processing the image data stored in the image memory and for 

2 0 displaying video images on a display device, the means for 
processing the image data comprising means for determining 
parameter sets of two or more motion models in accordance with a 
selection criterion, may advantageously benefit from the method 
according to the invention when the means for processing the image 

25 data further comprise means for block-wise evaluation of the 
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deviations between the current and the previous video image^_ and 
for selection of those blocks for use of the selection criterion, 
in which the value of the deviation exceeds a predeterminable 
threshold value. Conventional, digitally operating televisions and 
5 monitors may be operated in a simple manner in accordance with the 
method according to the invention, with an improvement of the 
quality of the image displayed. Devices in the sense mentioned 
above are, for example, the cathode ray tubes or dot matrix 
displays conventionally used in televisions and monitors. Other 

10 devices for visual display of digital image data are also feasible. 
[0016] According to the invention, a computer program product is 
suitable for interpolation between pairs of video image data sets, 
which this p roduct compris e s comprising , as input, a first and a 
second video image and, starting therefrom, computes parameter sets 

15 of two or more motion models and supplies motion data describing 

the displacement of image objects from the previous to the current 
image, while the image data of the two video images are compared 
with each other and only those parts of the image area in which 
there are significant differences between the two video images are 

2 0 taken into account in the computation of the parameter sets. The 
computer program product may be made available on various data 
carriersm such asm diskettes, CD-ROMs or the like, but also for 
transfer via computer networks (for example, Internet) . 

25 BRIEF DESCRIPTION OF THE DRAWINGS 
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[0017] These and other aspects of the invention are apparent 
from and will be elucidated with reference to the embodiments 
described hereinafter. 

[0018] In the drawings: 
5 [0019] Fig. 1 shows a selection of interesting image areas; 

[0020] Fig. 2 is a block diagram of a motion estimation method 
according to the invention; and 

[0021] Fig. 3 is a block diagram of a device according to the 
invention, for displaying video images. 

10 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0022] When determining the parameter sets for the motion models 
according to the invention, a selection criterion is applied to the 
selected image areas. The selection criterion consists of, for 
15 example, the evaluation of a match error £. This is computed as the 
sum of absolute differences of individual motion-compensated pixel 
intensities between a current and a previous video image in the 
following manner: 



20 



xel(n) 



25 



[0023] 



A summation is subsequently effected via image co- 



ordinates x = 



, comprised in a quantity I (n) of selected image 
areas. The absolute differences between the pixel intensities in 
the current video image an d the previous video image are added to 
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these image co-ordinates. F s (x,n) is the pixel intensity at the 
image co-ordinate x in a video image with a reduced raster. It has 
proved that-r in determining the parameter, the use of a resolution- 
reduced (sub-sampled) image is entirely sufficient. This 
5 advantageously leads to a considerable reduction of the number of 
computations. Due to the continuing index n, the number of the 
frame^_ and hence^ the instant within the video sequence^ is 
indicated. C 0 (x,n) indicates, for the image pair n f as a current 
video image and n-1 as a previous video image, the displacement 

10 vector at the image co-ordinate x in accordance with the motion 
model with the index o. W Q (x) represents a weighting factor with 
which it is taken into account which motion model o was assigned to 
the image co-ordinates x in earlier image data of the video 
sequence. A combination between the determination of the parameters 

15 and image segmentation can thereby be realized, which has 

advantages with a view to the temporal consistency of the motion 
estimation and the efficiency of the method. 

[0024] Starting from four parameters, displacement vectors can 
be computed by means of the following motion model: 

20 

- . , (s x {o,n) + x-d x (o,n)\ 
{s y (o,n) + yd y (o 9 n)) 

[0025] This is a simple linear first -order model with which 
translations and scalings can be described. The model is determined 
2 5 by the parameter setj^ 
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P 0 (n) = (s x (o, n\ d x (o y n), s y (o, n\ d y (o, n)J . 

The parameter set is determined in such a way that the above - 
5 mentioned match error for the corresponding motion model o assumes 
a minimal value. In the motion estimation method according to the 
invention, at least two motion models are used every time, one of 
which always has the zero vector as a parameter set so that the 
stationary image areas are described by this motion model with the 

10 displacement vector C 0 (x,n) = 0 . 

[0026] The next step in the motion estimation according to the 
invention is the image segmentation, i.e.^ assigning image areas to 
the motion models. To this end, the overall image area is initially 
subdivided into blocks. In practice, quadratic blocks of 8x8 pixels 

15 have prov e d proven to be suitable. For all image co-ordinates 

within the blocks at the position X , it then holds that xeB(X). 
For each block, a match error is again computed on a motion model 
o: 

2 0 e 0 (X, n) = X I F s (* + 0 " a )C 0 (*> «)> «) ~F s {x- aC 0 (x , n), n-l)\ 

xeB(X) 

[0027] The instant when the segmentation should be valid is 
determined by or. In the simplest case, that motion model o is 
assigned to the block X for which £ 0 {X,n) is minimal. The 
25 assignment is then filed in the segmentation mask M(X 9 n) . 
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[0028] According to the invention, the motion estimation method 
for determining the parameters of the motion models is limited to 
"interesting" image areas, given by the quantity I (n) . It is 
advantageous to fill the quantity I (n) with those blocks that are 
5 in poor conformity with the corresponding blocks in a previous 
image. This may be effected, for example, in accordance with the 
following prescription: 

I(n) = \x\e Q (X,n-l)ZT} 

10 

T is a predeterminable threshold value which fixes the extent of 
the deviation between two consecutive images as from which the 
parameters are determined in the relevant image area. 

15 [0029] Fig. 1 is a video image 1 showing a motorcyclist 2 riding 
on a street 3. The motorcyclist moves from left to right in the 
section of the image. The background, thus also the street 3, is 
stationary. In the Figure, the selection of " interesting" image 
areas can be recognized and are shown as white blocks 4. A motion 

20 model describing the motion of the motorcyclist 2 is assigned to 
the white blocks 4. The image background is stationary and is 
assigned to another corresponding motion model. 

[0030] Fig. 2 shows^ diagrammatical ly_^ the motion estimation 
procedure in accordance with the invention. Starting from a 
2 5 previous video image 6, a current video image 7 and a threshold 



ST14GOA0 . RED 



12 



0 «1 *' 



PHNL 000643 

value 8, image areas that are interesting for determining the 
parameter sets are selected in a first step 9 of the method 
described above. All of these image areas are provided with 
weighting factors 10 for a plurality of motion models and 
5 subsequently further processed in a step 11^ in which the 

parameters of the motion models are determined in accordance with a 
selection criterion. Starting from the completely determined motion 
models, the overall image area is then subdivided into blocks in a 
step 12, and the displacement vectors corresponding to the 

10 individual motion models are computed for each block. Subsequently, 
the image area is segmented in step 13, in which the blocks are 
assigned to the motion models. The assignments, which are included 
in the weighting 10 for the next image pair, are stored in a 
segmentation mask 14 which is then obtained. 

15 [0031] Fig. 3 shows^ diagrammatically^ the structure of a 

digitally operating device which may be, for example, a television 
or a video monitor. The device receives a video signal 2 0 which is 
stored and prepared in a digital image processing unit 21. To this 
end, the image processing unit comprises an image memory 22, a 

20 processor 23 and a program memory 24. These elements may also be at 
least partly combined in a discrete component. The processor 23 
runs through a program stored in the program memory 24, which this 
program controls controlling the image processing method according 
to the invention. A display unit 25 receives image data 26 prepared 

25 by the image processing unit 21 and generates a signal 2 7 therefrom 
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for driving a cathode ray tube 2 8 via which the video images are 
visually presented. 
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ABSTRACT-* - OF THE DISCLOSURE 

Th e inv e ntion relat e s to ln an object-oriented method of 
interpolation between video images^ — Two , two or more motion models 
5 are used for describing the displacement of image objects between 
two video images. The motion models are determined by parameter 
sets from which displacement vectors can be computed. One of the 
motion models is needed to take into account those parts of the 
image which are static, i.e.^ stationary. In accordanc e with th e 
10 inv e ntion; — tThe above method for determining the parameter sets for 
the motion models is limited to "interesting" image areas so as to 
save computing time. To this m e thod end , those parts of the image 
are selected in which the two video images are significantly 
distinct from each other. 

15 
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